Deep Gaussian Process Regression (DGPR)

A Gaussian Process Regression model is equivalent to an infinitely wide neural network with single hidden layer and similarly a DGP is a multi-layer neural network with multiple infinitely wide hidden layers [Neal, 1995]. DGPs employ a hierarchical structural of GP mappings and therefore are arguably more flexible, have a greater capacity to generalize, and are able to provide better predictive performance [Damianou, 2015]. Then it comes into my mind that why we would like to proceed to be deep and what are the benefits about being deep. It has been argued that the addition of non-linear hidden layers can also potentially overcome practical limitations of shallow GPs [Bui et al., 2016]. So what are the limitations exactly? Actually a GPR model is fully specified by a mean function E [·] and the covariance function cov [·, ·]. Conventionally we manually set the mean function to be 0. Then we can say that a GPR model is fully specified by its covariance function which also can be denoted as the kernel. Let us briefly examine the priors on functions encoded by some commonly used kernels 2 Expressing Structure with Kernels